Panoramic video, also known as 360-degree or immersive video, is becoming increasingly popular. In a typical panoramic video system, a user wearing a Virtual Reality (VR) headset can freely change his or her viewing direction. The user is often situated in the center of a virtual sphere, and the panoramic contents downloaded from video servers are projected onto the sphere (such as using Equirectangular projection). The user's viewport (or visible area) is determined by his or her viewing direction (such as in latitude/longitude) and the Field of View (FoV) of the VR headset in real time. The FoV defines the extent of the observable area, which is usually a fixed parameter of a VR headset. As shown in FIG. 1, the user wearing a VR headset can adjust his or her orientation by changing the pitch, yaw, and roll, which correspond to rotating along the X, Y, and Z axes, respectively.